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IMAGING SENSOR 



Background of the Invention 

The present invention proceeds from an imaging sensor of the 
species defined in the independent claim. 

WO 01/60662 Al has already disclosed an imaging sensor that 
is disposed in a vehicle. It is used therein for seat 
occupancy detection. 

Advantages of the Invention 

The imaging sensor according to the present invention having 
the features of the independent claim has, in contrast, the 
advantage that the imaging sensor monitors its functionality 
on the basis of its image signal. A safety requirement for 
an image recognition system of this kind is thereby met. The 
imaging sensor can be used for occupant detection, for 
determination of the occupant's posture, or for 
classification of the occupancy situation, but also for 
surroundings monitoring and also, for example, for rollover 
detection. In particular, an additional sensor suite for 
monitoring functionality is thus no longer necessary, or the 
demands on additional monitoring apparatuses can be reduced. 
Imaging sensors are applicable here, in particular, in 
safety- relevant applications such as occupant protection 
systems. It is specifically here that functionality of the 
imaging sensor is essential for functionality. 

The features and refinements set forth in the dependent 
claims make possible advantageous improvements to the 
imaging sensor described in the independent claim. 
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It is particularly advantageous that the imaging sensor has 
an evaluation unit that derives from the image signal at 
least one value that the evaluation unit compares with at 
least one limit value in order to monitor functionality. 
Empirical knowledge about measurement signal profiles can 
then be incorporated. It is possible in this context, in 
particular, to compare a limit value set that is stored in a 
memory associated with the imaging sensor. A system state, 
in particular, can be determined by the comparison with 
several limit values. Advantageously, that system state is 
then transmitted via an interface to further systems. That 
interface can be embodied as a two-wire interface, for 
example to a control unit, but it can also be embodied as a 
bus interface. Optical, electrical, or radio bus 
configurations, for example, can then be used therefor. 

It is further advantageous that the imaging sensor generates 
the image signal on the basis of at least one invariant 
pattern. That invariant image signal is then used for self- 
monitoring by being compared with an internal reference 
pattern. Naturally occurring invariant features of the 
surroundings; or invariant features automatically induced by 
a system, for example using an illumination module; or 
artificially induced invariant features of the surroundings, 
for example targets that are provided or generated by a test 
image process, can be used for this purpose. In the test 
image process, a simulated sensor signal is conveyed to the 
evaluation unit. The associated measurement signal is 
predefined. Discrepancies then result in an error message. 

It is further advantageous that the imaging sensor monitors 
its functionality on the basis of a profile of the image 
signal. This can be accomplished, for example, by way of a 
simple comparison of adjacent regions of the imaging sensor. 
A pattern comparison, i.e. a comparison with qualitative 
signal profiles, is also possible here. Trends or 
statistical parameters can be analyzed, or correlation 
methods can be applied to the image signal profile. Spectral 
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methods such as analysis of the Fourier spectrum, the 
wavelet spectrum, or the contrast spectrum can also, 
however, be applied here. 

5 It is additionally advantageous that if the imaging sensor 
has at least two image -producing sensors, it checks its 
functionality by comparing the output signals of those two 
image -producing sensors. The redundancy of a network of 
high-resolution sensors, for example an array or also a 
10 stereocamera, can thereby be utilized. Methods for analysis 
of the image signal profile are then applicable here as 
well. Utilization of a time-related redundancy is also 
possible here, by way of a time-related analysis of the 
sensor signal or analysis of recorded dynamic processes . 

15 

The self -monitoring of the imaging sensor can be performed 
in an initialization phase or also continuously or 
intermittently during operation. 

2 0 It is additionally advantageous that the imaging sensor is 

connectable to a diagnostic unit that activates the self- 
monitoring of the imaging sensor. That diagnostic unit can 
be disposed in the vehicle or also outside the vehicle, in 
order then to perform the self -monitoring via a radio 
25 connection. It is conceivable for an expanded test program 
also to be performed in the event of an activation by the 
diagnostic unit, since it is possible, for example, to 
transfer pattern files or also to perform long-term tests. 
It may furthermore be advantageous for the imaging sensor to 

3 0 be manually activatable for self -monitoring. The imaging 

sensor then has corresponding operating elements or 
interfaces for that purpose, which initiate the self- 
monitoring by way of an actuation of a device. 

35 The imaging sensor can be configured, in particular, in 

depth- imaging fashion, i.e. two image sensors, for example, 
are used in order to obtain depth resolution of an object. A 
matrix or an array of image sensors can also be used for 
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this. Also conceivable is a depth- imaging sensor that 
operates according to different physical principles, for 
example the time-of -flight principle or the principle of 
structured illumination. 

For self -monitoring but also for other purposes, it may be 
advantageous to have an illumination apparatus that is 
associated with the imaging sensor. 

Drawing 

Exemplified embodiments of the invention are depicted in the 
drawings and are explained in more detail in the description 
below. 

In the drawings : 

Figure 1 is a first block diagram of the imaging sensor 

according to the present inventions- 
Figure 2 is a second block diagram of the imaging sensor 

according to the present inventions- 
Figure 3 is a third block diagram of the imaging sensor 

according to the present inventions- 
Figure 4 is a fourth block diagram; and 
Figure 5 is a fifth block diagram. 
Description 

Highly developed high- resolution image -producing or depth- 
image -producing measurement systems are of increasing 
interest for applications in automotive engineering. Video- 
based assistance systems and safety systems are envisioned 
here as particular applications. The more responsibility 
that is to be transferred from human beings, the more 
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reliable such a measurement system must be. In that context, 
the ability of the system to detect a failure and initiate 
suitable actions is also very important. The present 
invention proposes an imaging sensor of this kind that has 
5 this capability for self -monitoring, that imaging sensor 
being built into a motor vehicle. The essence of the 
invention is the integration of this self -monitoring 
functionality into a high-resolution image -producing or 
depth- image -producing measurement system. 

10 

Since such measurement systems for measured value generation 
have at least one high-performance evaluation unit, a self- 
monitoring functionality is implemented by the fact that 
signal processing methods are used, by way of the evaluation 

15 unit, to ascertain, from the sensor signals themselves, 
variables that allow conclusions as to the functional 
capability of the sensor and the measurement system. 
Previous knowledge and empirical knowledge about signal 
profiles is evaluated in suitable fashion. In the simplest 

20 case, a parameter that has been derived from the image 

signal is compared with a limit value or a limit value set 
that is stored in a memory associated with the imaging 
sensor. 

25 Another possibility is to perform an evaluation of the 

system status on the basis of several different variables. 
If limit values are exceeded or if a limited functionality 
(up to and including sensor failure) is identified in 
another fashion, a corresponding status report is then 

3 0 transmitted via a suitable interface, but at least the 

failure of the imaging sensor is reported. Otherwise the 
functional capability of the imaging sensor is transmitted 
via that interface. The self -monitoring can be performed 
during the initialization phase of the imaging sensor, at 

35 certain points in time or continuously. The self -monitoring 
can also be activated externally, i.e. by way of a higher- 
order system such as a diagnostic unit, or manually. It is 
conceivable, in the case of activation by way of a 
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diagnostic system, also to perform an expanded test program, 
since it is possible, for example, to transfer pattern files 
or also to perform long-term tests. 

5 Figure 1 shows the imaging sensor according to the present 

invention in a first block diagram. Physical process 10 (the 
scene) is imaged by sensor 12 as an image signal. Sensor 12, 
together with a processing unit 13, constitutes a 
measurement system. The image signal generated by sensor 12 
10 is prepared and processed by processing unit 13 . The 

measurement signal, i.e. the image signal, is transferred 
via a first interface 14 to further systems, for example to 
a control unit for occupant detection. 

15 The status of the imaging sensor which is- presented here, 
and which was also determined on the basis of the image 
signal, is transferred via a further interface 15. As stated 
above, the status of the imaging sensor, i.e. its self- 
monitoring, is performed either by utilizing previous 

2 0 knowledge about its invariant patterns, or empirical 
knowledge about measurement signal profiles or the 
redundancy of a network of sensors, or by utilizing time- 
related redundancy. Interfaces 14 and 15 can also be 
combined into one interface and are then separated only 

25 logically. The interfaces here can be two-wire interfaces or 
also interfaces to a bus system. 

Figure 2 shows an imaging sensor that has more than one 
sensor for image acquisition and is thus also configured for 

30 depth-image production. Three sensors 22 through 24 are 

depicted here by way of example, although it is possible to 
use only two sensors or also more sensors. Measurement 
system 21 is therefore constituted by sensors 22 through 24 
and processing unit 25. Physical process 20 (the scene) is 

35 imaged by sensors 22-24. Processing unit 25 receives the 
image signals of image sensors 22 through 24, processes 
them, and then, as a function of the evaluation of those 
image signals, conveys signals to interfaces 26 and 27 in 
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order to transfer on the one hand the status of the imaging 
sensor and on the other hand the measurement signal itself. 
Sensors 2 2 through 24 can be connected to individual 
interface modules of processing unit 25, but they can also 
be connected to processing unit 25 via a multiplexer or an 
internal bus . The imaging sensor can be embodied in one 
physical unit in which interfaces 26 and 27 are also 
integrated. Another possibility, however, is that no housing 
exists for the entirety of these components, and instead 
they are disposed in distributed fashion. Processing unit 25 
then, as described above, performs the analysis of the image 
signal in order to perform the self -monitoring of the 
imaging sensor. 

Figure 3 shows the imaging sensor according to the present 
invention in a third block diagram. Two sensors are present 
here, as video cameras 31 and 32 that are connected to a 
processing unit 33 . The latter has a program 34 for sensor 
data processing and a program 35 for self -monitoring. Self- 
monitoring 35 is also performed on the image signals of 
video cameras 31 and 32. In addition, processing unit 33 
controls an illumination unit or a signal generator 36, for 
example in order to perform the self -monitoring by comparing 
self -induced patterns with their internal representation. 
Processing unit 33 is furthermore connected to interfaces 37 
and 38 that serve to transfer respectively the measurement 
signal, i.e. the image or depth image, and the status or the 
result of the self -monitoring • Measurement system 3 0 thus 
comprises video cameras 31 and 32, processing unit 33, and 
illumination unit 36. The overall imaging sensor is 
supplemented with interfaces 37 and 38. Sensors 31 and 32 
are embodied here as video cameras . The output signals are 
conveyed to evaluation unit 33, which performs suitable 
processing steps such as image processing, correlation 
methods, or triangulation in order to generate the spatial 
data. This processing unit 33 also, however, perfomnas 
suitable processes for self -monitoring of the measurement 
system. In this exemplified embodiment, the output signals 
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of the stereoscopic video measurement system are the image, 
the depth image, and the status signal of measurement system 
30. 

The table below lists potential problems that can result in 
limited functionality of the embodied measurement system. 
Columns 2 and 3 contain the appropriate data and signal 
processing methods for identifying the limited 
functionality . 
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Problem 


Data analyzed 


Selection of self -monitor ing 

methods 


Sensor is 
(partly) 
obstructed 


Grayscale 
image of 
sensor 1 


Utilize previous knowledge about 
invariant patterns : 

• Naturally occurring invariant 
features of the surroundings . 

• Invariant features induced 
automatically (e.g. using an 
illumination module) by the 
system. 

Utilize time -related redundancy: 

• Time-related analysis of sensor 
signal . 

• Analysis of recorded dynamic 
processes . 


Grayscale 
image of 
sensor 2 


see Grayscale image of sensor 1 


Grayscale 
image s of 
sensors 1 and 
2 


Utilize empirical knowledge about 

measurement signal profiles : 

• Analysis of statistical 

parameters 

Utilize redundancy of a network of 
high- resolution sensors: compare 
various individual sensor signals 
of the sensor network 


Depth image 


see Grayscale image of sensor 1 


Decalibrat 
ion 

detection 


Grayscale 
image of 
sensor 1 


Utilize previous knowledge about 
invariant patterns : 

• Naturally occurring invariant 
features of the surroundings. 

• Invariant features induced 
automatically (e.g. using an 
illumination module) by the 
system. 


Grayscale 
image of 


see Grayscale image of sensor 1 
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sensor 2 




Grayscale 
images of 
sensors 1 and 
2 


Utilize redundancy of a network of 
high-resolution sensors: compare 
various individual sensor signals 
of the sensor network 


Errors in 
brightness 
setting 


Grayscale 
image of 
sensor 1 


Utilize empirical knowledge about 
measurement signal profiles : 

• Simple comparison of adjacent 
regions of sensor 

• Comparison with limit values 

• Comparison with cpialitative 
signal profiles 

• Analysis of trends 

• Analysis of statistical 
parameters 

• Analysis of further spectral 
properties 

Utilize time-related redundancy: 

• Time-related analysis of sensor 
signal . 

• Analysis of recorded dynamic 
processes . 


Grayscale 
image of 
sensor 2 


see Grayscale image of sensor 1 


Grayscale 
image s of 
sensors 1 and 
2 


Utilize redundancy of a network of 
high-resolution sensors: compare 
various individual sensor signals 
of the sensor network 


Errors in 
image 
s ha r pne s s 
( def ocusin 

g) 


Grayscale 
image of 
sensor 1 


Utilize empirical knowledge about 
measurement signal profiles: 

• Analysis of statistical 
parameters 

• Analysis of contrast spectrum 

• Analysis of further spectral 
properties 

Utilize time-related redundancy: 
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• Time-related analysis of sensor 
signal . 

• Analysis of recorded dynamic 
processes . 


Grayscale 
image of 
sensor 2 


see Grayscale image of sensor 1 


Grayscale 
image s of 
sensors 1 and 
2 


Utilize redundancy of a network of 
high-resolution sensors: compare 
various individual sensor signals 
of the sensor network 



A stereoscopic video-based measurement system may be 
regarded as a typical example of a high-resolution image- 
producing or depth- image -producing measurement system to 
3 0 which many of the aforementioned signal -processing or 
pattern-recognition methods can be applied for self- 
monitoring. A largely independent generation of the 
individual sensor signals, in particular, should make 
possible a powerful self -monitoring functionality. 

35 

Figure 4 is a fourth block diagram of the imaging sensor 
according to the present invention. A video sensor system 40 
has a camera system 42 that is connected on the one hand to 
an image preprocessor 43 and on the other hand to an output 
40 of the video sensor system. Image preprocessor 43 is 

connected to a comparison unit 44 that is connected via an 
output to an evaluation unit 45 and via an input to a device 
for structured illumination. Evaluation unit 45 supplies the 
sensor status via an output of video sensor system 40. 

45 

Device 41 for structured illumination radiates structured 
light, constituting a reference pattern, into the 
surroundings of video sensor system 40, in particular onto a 
surface 47 on which the reference pattern is imaged. That 
50 surface is then referred to as reference surface 47. 

Reference surface 47 is rigid and stationary. Conceivable 
reference surfaces are object surfaces present in the 
sensing region of video sensor system 40, for example a roof 
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liner when the sensor is used to monitor a motor vehicle 
interior. Also possible, however, are special calibration 
elements that, for example, are mounted in a defined 
location and orientation during the manufacturing process. 

Camera system 42, which can comprise one or more cameras, 
senses the reference pattern on reference surface 47. The 
two-dimensional camera images are compared in comparison 
unit 44 to the reference pattern; the two-dimensional camera 
images can also be ones that were prepared in the optional 
image preprocessor 43. That preparation can be a filtration. 
Comparison unit 44 can have a memory unit in which, for 
example, the reference patterns are stored, if they are not 
conveyed from the unit for structured illumination in the 
form of a signal. The sensor status is then ascertained in 
evaluation unit 45 on the basis of the results of comparison 
unit 44 . The fact that the sensor is obstructed or 
unobstructed and/or the sensor optics are focused or 
unfocused and/or the optical image is distorted or 
undistorted can be regarded, for example, as the sensor 
status. Evaluation unit 45 can also contain a memory in 
which, for example, specific patterns can be stored that are 
created upon comparison of the reference pattern with the 
camera images of a faulty video sensor system. 

With the device described above it is possible, for example, 
to identify a defocusing of the sensor by analyzing whether 
a sharp image of the reference pattern is present in the 
camera image. Complete or partial obstruction of the sensor 
can furthermore be detected by checking whether the 
reference pattern is imaged in complete and undistorted 
fashion in the camera image. Distortions of the optical 
image result in distorted imaging of the reference pattern 
in the camera image, and can thus be identified using 
comparison unit 44 and evaluation unit 45. Further errors 
that can detected with this system are soiling of the 
optics, and misalignment of the absolute calibration. Here 
the resulting shift and distortion of the reference pattern 
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is detected. An initial calibration or post -calibration can 
in fact be performed on these data. 

Device 41 for structured illumination can be integrated into 
5 the video sensor. A device for structured illumination 

separate from the video sensor is also possible, however, 
especially in the manufacturing process and for checking the 
video sensor at a repair shop. In this case a defined 
orientation of the device for structured illumination with 
10 respect to the video sensor is necessary. With this 

procedure, therefore, in the simplest case the video image 
of the structured illumination is interpreted directly. An 
evaluation of a three-dimensional image is also possible. 

15 Figure 5 is a further block diagram. A video sensor system 
50 has two cameras 52 and 53 that supply their respective 
camera images to a unit 54 for determining three-dimensional 
measured values. A three-dimensional point cloud then 
results therefrom and is conveyed on the one hand to a 

20 signal preprocessor 55 and on the other hand to an output of 
video sensor system 50. Signal preprocessor 55 is connected 
to a comparison unit 56 to which a device 51 for structured 
illumination is also connected. Comparison unit 56 is 
connected via a data output to an evaluation unit 57 that in 

25 turn outputs the sensor status. 

Device 51 for structured illumination illuminates a 
reference surface in surroundings 5 8 of the video sensor. 
The reflected pattern is acquired by cameras 52 and 53 . Unit 

30 54 determines the three-dimensional point cloud from the 

camera images based on the stereo measurement principle. In 
addition to determination of the three-dimensional point 
cloud, the two-dimensional camera images can also be 
evaluated directly. A further possibility lies in evaluating 

35 the three-dimensional measured values using a range sensor 
that operates on the time-of -flight principle. 
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